Digital camera modules are currently being incorporated into a variety of host devices. Such host devices include cellular telephones, personal data assistants (PDAs), computers, and so forth. Consumer demand for digital camera modules in host devices continues to grow.
There is an ever-increasing demand from host device manufacturers to include higher-performance cameras, with better capabilities such as optical zoom, improved low-light performance and higher image quality. To tackle this demand, new camera systems have been proposed recently. Such camera systems include two cameras aligned to look in the same direction, with partially or fully overlapping fields of view (FOVs) and are referred to herein as “dual-camera” systems (or “dual-aperture camera” systems, with two apertures A and B), see e.g. international patent applications PCT/IB2014/062180, PCT/IB2014/063393 and PCT/IB2016/050844. The two cameras may have similar FOVs or very different FOVs, depending on the lenses used in each. It has been shown (see e.g. PCT/IB2014/062180 and PCT/IB2014/063393) that the images from the two cameras may be “stitched” or “fused” together according to a dedicated algorithm to form a composite image, with improved resolution, improved noise performance and improved image quality (at least for some part of the composite image field of view). The image stitching or image fusion algorithm can be implemented in software, running on an application processor (AP), or in hardware (hard-wired implementation).
It has also been shown (see e.g. co-owned U.S. Pat. No. 9,185,291) that some dual-camera systems, such as ones that provide high-quality zoom during preview or video recording or as ones that provide enhanced low light performance, may include a transition between one camera stream to the other camera stream in order to generate an output stream of frames, which is used in turn to show the preview or to record the video. This transition takes place at a certain zoom factor (ZF) when zooming in and out. In some cases, it is beneficial to keep the transition between the two cameras as smooth as possible—for example, in case the two cameras in the dual-camera system have different FOVs and where the dual-camera system implements continuous zooming between the two cameras. A smooth transition is a transition in which the user does not notice the transition point between the two cameras. A smooth transition should be smooth in time and space, namely continuous in both aspects.
Furthermore, it is known that some dual-camera systems may include calculation of a depth map from the two camera frames. A depth map is a map of the pixels in the frame, in which each object's relative distance in a scene is determined from the spatial shift of the object's image between the two frames. In some embodiments, the depth map requires a registration step between the frames from the two cameras. A registration step is a step in which a match is found between pixels in the two images that correspond to the same object in the scene, and a disparity value that represents the offset between the location on the sensor of the two corresponding pixels is assigned to each pair of matched pixels to form a “dense disparity map”. Alternatively, a registration step may include extracting features from the two frames, finding matches between features corresponding to the same object in the scene and calculating from the matched features a “sparse depth map”. The depth map may be calculated on a preview or video stream, or on a snapshot image.
For the three applications mentioned above (fusion of two captured images, transition between two streams of frames and creating a depth map from two camera frames), the synchronization of the acquisition time of the frames is an important requirement and common practice. For example, when registering information between two frames from the two cameras, any object motion in the scene or motion of the dual-aperture camera may result in registration errors if the frame acquisition time is not synchronized within a certain period of time (e.g. less than 3-5 msec). The registration errors can lead to wrong depth estimations when calculating a depth map. In smooth transition, lack of synchronization in time between pairs of frames from the two cameras may lead to a noticeable discontinuity when switching from one camera to the other.
A known in the art synchronization method between two camera sensors includes sending a synchronization signal every frame from one sensor, denoted “master sensor”, to the second sensor, denoted “slave sensor”. This method requires the two cameras to output the stream of frames at approximately the same rate to stay synchronized (for example, both sensors will output the frames at a rate of 30 fps).
Apart from maintaining synchronization, there are other benefits to keeping the two cameras streaming in parallel at all times (even when only one camera is actually used to generate the output image or frame): first, it is desired to maintain accurate information of focus, white balance and light gain level (known as “3A information”) for both cameras, even when one is not used, in order to be able to use information from the not used camera with as small a latency as possible. If one camera is set to be in “standby” mode and does not stream frames, it may take up to several seconds until white balance, exposure and focus converge to values that match the scene when configuring the camera to start streaming frames. This time may hinder user experience and may prevent smooth transition from one camera to the other, for example when zooming-in or zooming-out, or for example when switching from regular light mode to low light mode. Second, registration may be required to be maintained at all times, for example for the purpose of calculating a depth map of the scene from the two images. However, running two camera sensors in parallel carries the penalty of doubling power consumption.
In summary, to enable fast output switching between one aperture (camera) and another aperture (camera) in a dual-camera, both camera need to be operative and synchronized. This creates a power consumption problem, since keeping two cameras fully operational results in doubling the combined camera power consumption in comparison with that of a single camera system. At present, there are no satisfactory solutions to this power consumption problem.